Low-frequency effects haptic conversion system

ABSTRACT

A system is provided that produces haptic effects. The system receives an audio signal that includes a low-frequency effects audio signal. The system further extracts the low-frequency effects audio signal from the audio signal. The system further converts the low-frequency effects audio signal into a haptic signal by shifting frequencies of the low-frequency effects audio signal to frequencies within a target frequency range of a haptic output device. The system further sends the haptic signal to the haptic output device, where the haptic signal causes the haptic output device to output one or more haptic effects.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 14/277,870, filed on May 15, 2014 (the disclosureof which is hereby incorporated by reference), where U.S. patentapplication Ser. No. 14/277,870 claims priority of U.S. ProvisionalPatent Application Ser. No. 61/824,442, filed on May 17, 2013 (thedisclosure of which is hereby incorporated by reference).

FIELD

One embodiment is directed generally to a device, and more particularly,to a device that produces haptic effects.

BACKGROUND

Electronic device manufacturers strive to produce a rich interface forusers. Conventional devices use visual and auditory cues to providefeedback to a user. In some interface devices, kinesthetic feedback(such as active and resistive force feedback) and/or tactile feedback(such as vibration, texture, and heat) is also provided to the user,more generally known collectively as “haptic feedback” or “hapticeffects”. Haptic feedback can provide cues that enhance and simplify theuser interface. Specifically, vibration effects, or vibrotactile hapticeffects, may be useful in providing cues to users of electronic devicesto alert the user to specific events, or provide realistic feedback tocreate greater sensory immersion within a simulated or virtualenvironment.

In order to generate vibration effects, many devices utilize some typeof actuator or haptic output device. Known haptic output devices usedfor this purpose include an electromagnetic actuator such as anEccentric Rotating Mass (“ERM”) in which an eccentric mass is moved by amotor, a Linear Resonant Actuator (“LRA”) in which a mass attached to aspring is driven back and forth, or a “smart material” such aspiezoelectric, electro-active polymers or shape memory alloys. Hapticoutput devices also broadly include non-mechanical or non-vibratorydevices such as those that use electrostatic friction (“ESF”),ultrasonic surface friction (“USF”), or those that induce acousticradiation pressure with an ultrasonic haptic transducer, or those thatuse a haptic substrate and a flexible or deformable surface, or thosethat provide projected haptic output such as a puff of air using an airjet, and so on.

Within the film and television industries, improved multi-channel audiosystems have been developed in order to provide a more engagingexperience for viewers. For example, an analog six channel surroundsound multichannel audio system referred to as “5.1” was first developedby Dolby Laboratories, Inc., for 70 mm theatrical film screenings,providing three screen channels, two surround channels and alow-frequency enhancement channel. Later, a digital version of 5.1multi-channel audio referred to as “Dolby Digital” was developed for 35mm film, and, subsequently, a similar 5.1 system was developed by DTS,Inc. Since then, various multi-channel audio formats have been developedto include 6.1, or 7.1 source material and output up to 11.1 channelsand beyond, and multi-channel audio is now included with almost all DVD,Blu-ray, broadcast and streaming video content for home viewing.

With the development of recent high resolution mobile devices such assmart phones and tablets, users are now able to view high definitionaudio and video on a handheld device that traditionally could only beseen in movie theaters, television or home theater systems. Withhaptically enabled mobile devices, experience has shown that contentviewing is sufficiently enhanced, and viewers like it, if there is ahaptic content component in addition to the audio and video contentcomponents.

SUMMARY

One embodiment is a system that produces haptic effects. The systemreceives an audio signal that includes a low-frequency effects audiosignal. The system further extracts the low-frequency effects audiosignal from the audio signal. The system further converts thelow-frequency effects audio signal into a haptic signal by shiftingfrequencies of the low-frequency effects audio signal to frequencieswithin a target frequency range of a haptic output device. The systemfurther sends the haptic signal to the haptic output device, where thehaptic signal causes the haptic output device to output one or morehaptic effects.

BRIEF DESCRIPTION OF THE DRAWINGS

Further embodiments, details, advantages, and modifications will becomeapparent from the following detailed description of the preferredembodiments, which is to be taken in conjunction with the accompanyingdrawings.

FIG. 1 is a block diagram of a haptically-enabled system according toone embodiment of the invention.

FIG. 2 is a cut-away perspective view of an LRA implementation of ahaptic actuator according to one embodiment of the invention.

FIG. 3 is a cut-away perspective view of an ERM implementation of ahaptic actuator according to one embodiment of the invention.

FIGS. 4A-4C are views of a piezoelectric implementation of a hapticactuator according to one embodiment of the invention.

FIG. 5 is a view of a haptic device using electrostatic friction (“ESF”)according to one embodiment of the invention.

FIG. 6 is a view of a haptic device for inducing acoustic radiationpressure with an ultrasonic haptic transducer according to oneembodiment of the invention.

FIG. 7 is a view of a haptic device using a haptic substrate andflexible or deformable surface according to one embodiment of theinvention.

FIGS. 8A-8B are views of a haptic device using ultrasonic surfacefriction (“USF”) according to one embodiment of the invention.

FIG. 9 is an example 5.1 multi-channel audio surround configurationaccording to one embodiment of the invention.

FIG. 10 is human auditory system equal-loudness chart according to oneembodiment of the invention.

FIGS. 11A-11B are audio spectrograms showing a shifted, amplified andcompressed audio signal according to one embodiment of the invention.

FIGS. 12A-12B are Blackman-Harris windows showing a shifted audio signalaccording to one embodiment of the invention.

FIG. 13 is a quality of experience chart and corresponding low-frequencyeffects (“LFE”) haptics chart according to one embodiment of theinvention.

FIGS. 14A-14C illustrate a quality of experience chart and correspondingLFE haptics chart according to one embodiment of the invention.

FIGS. 15A and 15B illustrate a quality of experience chart andcorresponding LFE haptics chart according to one embodiment of theinvention.

FIGS. 16A and 16B illustrate a quality of experience chart andcorresponding LFE haptics chart according to one embodiment of theinvention.

FIGS. 17A-17C illustrate a quality of experience chart and correspondingLFE haptics chart according to one embodiment of the invention.

FIG. 18 is an immersiveness summary chart according to one embodiment ofthe invention.

FIG. 19 is a quality of experience summary chart according to oneembodiment of the invention.

FIG. 20 is a flow diagram for converting an LFE audio signal into ahaptic signal according to an embodiment of the invention.

FIG. 21 is a flow diagram for converting an LFE audio signal into ahaptic signal according to another embodiment of the invention.

FIG. 22 is a flow diagram for converting an LFE audio signal into ahaptic signal according to another embodiment of the invention.

FIG. 23 is a flow diagram for encoding a haptic signal within an LFEaudio signal according to an embodiment of the invention.

FIG. 24 is a flow diagram for decoding a haptic signal from alow-frequency effect signal according to an embodiment of the invention.

FIG. 25 is a flow diagram for converting an LFE audio signal into aplurality of haptic signals according to an embodiment of the invention.

FIGS. 26A-26D are screen views of example foreground and backgroundhaptic applications according to one embodiment of the invention.

FIGS. 27A-27B are display graphs of example multiple data channels ofhaptic feedback according to one embodiment of the invention.

FIG. 28 is a flow diagram for displaying multiple data channels ofhaptic feedback for priority based haptic events according to oneembodiment of the invention.

FIG. 29 is a flow diagram for displaying multiple data channels ofhaptic feedback for priority based haptic events according to oneembodiment of the invention.

FIG. 30 is a flow diagram for converting an LFE audio signal into ahaptic signal according to another embodiment of the invention.

DETAILED DESCRIPTION

One embodiment is a system that extracts a low-frequency effects (“LFE”)audio signal from a source audio signal and converts the extracted LFEaudio signal into a haptic signal, where the haptic signal causes ahaptic output device, such as an actuator, to output one or more hapticeffects. By converting the extracted LFE audio signal into the hapticsignal, the audio signal frequencies can be shifted to frequencieswithin a target frequency range of the haptic output device.Alternatively, by converting the extracted LFE audio signal into thehaptic signal, the audio signal pitch can be shifted to a pitch within atarget pitch range of the haptic output device. In embodiments where theconversion of the extracted LFE audio signal into the haptic signal is“offline,” the haptic signal can be encoded and stored within a format,such as a storage device, or the haptic signal can be encoded and storedwithin the LFE audio signal that is included within the source audiosignal. Alternatively, in embodiments where the conversion of theextracted LFE audio signal into the haptic signal is “online,” thehaptic signal can be sent to the haptic output device in real-time ornear real-time, where one or more haptic effects are output in real-timeor near real-time. In certain embodiments, the source audio signal canbe replaced by another type of input signal, and the LFE audio signalcan be replaced by another type of LFE signal. Further, in an alternateembodiment, the system can convert the extracted LFE audio signal into aplurality of haptic signals (either simultaneously or sequentially),where the haptic signals cause a plurality of haptic output devices(e.g., actuators) to output one or more haptic effects. In thisalternate embodiment, each haptic output device can have a distincttarget frequency range, and, for each conversion, the audio signalfrequencies of the extracted LFE audio signal can be shifted tofrequencies within each target frequency range of each haptic outputdevice.

As described below, a LFE channel is any audio channel that has beenencoded with an audio spectrum substantially less than the full spectrumof human auditory perception. An LFE track is typically used to encodelow frequency information with audible frequencies in the range of 20Hz-120 Hz, but may include any other audio signal with a limitedfrequency range such as a high frequency range sent to a “tweeter”speaker or a mid-range frequency range sent to a “squawker” speaker, ora low frequency range sent to a “woofer” speaker. Various popular audioencodings support this type of track. It is commonly referred to as the‘0.1’ in a 5.1, 7.1 or 11.1 surround sound audio track. The DVD andBlu-Ray standards specify the inclusion of an LFE track in consumerencodings. The LFE track is also used in surround sound video games,particularly, but not necessarily, those on 3rd generation gamingconsoles such as Sony PS3, or Microsoft XBOX 360. In this case the LFEtrack is generated in real time through the use of an audiospatialization engine that synthesizes the 5.1 surround audio channelsin real time depending on the game state such as the orientation of theplayer.

Traditional automated haptic playback architectures can provide hapticoutput based on an audio signal. Authored content by a haptic effectsprofessional can in many cases provide for a more compelling contentviewing experience than automated haptic generation from an audiosignal, but it is expensive because it requires a relatively largeamount of time to author the hundreds or thousands of haptic effects forlong-form content. Therefore, there is a need for an improved system ofautomatically providing haptic effects from an LFE audio signal thatdoes not require the same amount of time as authored haptic long-formcontent.

Because an LFE audio signal is typically intended to be reproduced usingan audio output device configured to handle low-frequency signals, thedata contained within the LFE audio signal is particularly well-suitedto conversion to a haptic signal. For example, an LFE audio signal isalready filtered and mixed to contain low-frequency (orlimited-frequency) audio, but at a full bit rate. Further, an LFE audiosignal is typically leveled by a content producer to have more correctamplitude with respect to other audio channels of an audio signal. Inaddition, an LFE audio signal typically contains the creative contentcomponent that is most natural for haptic-content experiences.

Unlike normal stereo audio data, LFE audio data translates naturally toa haptic signal. In stereo audio (or full range audio), it is generallynecessary to filter and extract out those signal components that aremost appropriate for haptic rendering. However, this can be achallenging operation which can result in inconsistent hapticexperiences. The conversion of an LFE audio signal into a haptic signalis further described below in greater detail.

FIG. 1 is a block diagram of a haptically-enabled system 10 according toone embodiment of the invention. System 10 includes a touch sensitivesurface 11 or other type of user interface mounted within a housing 15,and may include mechanical keys/buttons 13. Internal to system 10 is ahaptic feedback system that generates vibrations on system 10. In oneembodiment, the vibrations are generated on touch surface 11.

The haptic feedback system includes a processor 12. Coupled to processor12 is a memory 20 and an actuator drive circuit 16, which is coupled toa haptic actuator 18. In certain embodiments, actuator 18 can bereplaced with another type of haptic output device. Processor 12 may beany type of general purpose processor, or could be a processorspecifically designed to provide haptic effects, such as anapplication-specific integrated circuit (“ASIC”). Processor 12 may bethe same processor that operates the entire system 10, or may be aseparate processor. Processor 12 can decide what haptic effects are tobe played and the order in which the effects are played based on highlevel parameters. In general, the high level parameters that define aparticular haptic effect include magnitude, frequency and duration. Lowlevel parameters such as streaming motor commands could also be used todetermine a particular haptic effect. A haptic effect may be considereddynamic if it includes some variation of these parameters when thehaptic effect is generated or a variation of these parameters based on auser's interaction.

Processor 12 outputs the control signals to drive circuit 16 whichincludes electronic components and circuitry used to supply actuator 18with the required electrical current and voltage to cause the desiredhaptic effects. System 10 may include more than one actuator 18, andeach actuator may include a separate drive circuit 16, all coupled to acommon processor 12. Memory device 20 can be any type of storage deviceor computer-readable medium, such as random access memory (“RAM”) orread-only memory (“ROM”). Memory 20 stores instructions executed byprocessor 12. Among the instructions, memory 20 includes an actuatordrive module 22 which are instructions that, when executed by processor12, generate drive signals for actuator 18 while also determiningfeedback from actuator 18 and adjusting the drive signals accordingly.In certain embodiments, actuator drive module 22 can be a low-frequencyeffects conversion module that can generate drive signals based onlow-frequency effects audio signals. These drive signals are alsoidentified as haptic signals. The functionality of module 22 isdiscussed in more detail below. Memory 20 may also be located internalto processor 12, or any combination of internal and external memory.

Touch surface 11 recognizes touches, and may also recognize the positionand magnitude or pressure of touches on the surface, such as the numberof touches, the size of the contact points, pressure, etc. The datacorresponding to the touches is sent to processor 12, or anotherprocessor within system 10, and processor 12 interprets the touches andin response generates haptic effect signals. Touch surface 11 may sensetouches using any sensing technology, including capacitive sensing,resistive sensing, surface acoustic wave sensing, pressure sensing,optical sensing, etc. Touch surface 11 may sense multi-touch contactsand may be capable of distinguishing multiple touches that occur at thesame time. Touch surface 11 may be a touchscreen that generates anddisplays images for the user to interact with, such as keys, dials,etc., or may be a touchpad with minimal or no images.

System 10 may be a handheld device, such as a cellular telephone, PDA,computer tablet, gaming console, wearable device, etc. or may be anyother type of device that provides a user interface and includes ahaptic effect system that includes one or more ERMs, LRAs, electrostaticor other types of actuators. The user interface may be a touch sensitivesurface, or can be any other type of user interface such as a mouse,touchpad, mini-joystick, scroll wheel, trackball, game pads or gamecontrollers, etc. In embodiments with more than one actuator, eachactuator may have a different output capability in order to create awide range of haptic effects on the device. Each actuator may be anytype of haptic actuator or a single or multidimensional array ofactuators.

FIG. 2 is a cut-away side view of an LRA implementation of actuator 18in accordance to one embodiment. LRA 38 includes a casing 25, amagnet/mass 27, a linear spring 26, and an electric coil 28. Magnet 27is mounted to casing 25 by spring 26. Coil 28 is mounted directly on thebottom of casing 25 underneath magnet 27. LRA 38 is typical of any knownLRA. In operation, when current flows through coil 28 a magnetic fieldforms around coil 28 which in interaction with the magnetic field ofmagnet 27 pushes or pulls on magnet 27. One current flowdirection/polarity causes a push action and the other a pull action.Spring 26 controls the up and down movement of magnet 27 and has adeflected up position where it is compressed, a deflected down positionwhere it is expanded, and a neutral or zero-crossing position where itis neither compressed or deflected and which is equal to its restingstate when no current is being applied to coil 28 and there is nomovement/oscillation of magnet 27.

For LRA 38, a mechanical quality factor or “Q factor” can be measured.In general, the mechanical Q factor is a dimensionless parameter thatcompares a time constant for decay of an oscillating physical system'samplitude to its oscillation period. The mechanical Q factor issignificantly affected by mounting variations. The mechanical Q factorrepresents the ratio of the energy circulated between the mass andspring over the energy lost at every oscillation cycle. A low Q factormeans that a large portion of the energy stored in the mass and springis lost at every cycle. In general, a minimum Q factor occurs withsystem 10 is held firmly in a hand due to energy being absorbed by thetissues of the hand. The maximum Q factor generally occurs when system10 is pressed against a hard and heavy surface that reflects all of thevibration energy back into LRA 38.

In direct proportionality to the mechanical Q factor, the forces thatoccur between magnet/mass 27 and spring 26 at resonance are typically10-100 times larger than the force that coil 28 must produce to maintainthe oscillation. Consequently, the resonant frequency of LRA 38 ismostly defined by the mass of magnet 27 and the compliance of spring 26.However, when an LRA is mounted to a floating device (i.e., system 10held softly in a hand), the LRA resonant frequency shifts upsignificantly. Further, significant frequency shifts can occur due toexternal factors affecting the apparent mounting weight of LRA 38 insystem 10, such as a cell phone flipped open/closed or the phone heldtightly.

FIG. 3 is a cut-away perspective view of an ERM implementation ofactuator 18 according to one embodiment of the invention. ERM 300includes a rotating mass 301 having an off-center weight 303 thatrotates about an axis of rotation 305. In operation, any type of motormay be coupled to ERM 300 to cause rotation in one or both directionsaround the axis of rotation 305 in response to the amount and polarityof voltage applied to the motor. It will be recognized that anapplication of voltage in the same direction of rotation will have anacceleration effect and cause the ERM 300 to increase its rotationalspeed, and that an application of voltage in the opposite direction ofrotation will have a braking effect and cause the ERM 300 to decrease oreven reverse its rotational speed.

One embodiment of the present invention provides haptic feedback bydetermining and modifying the angular speed of ERM 300. Angular speed isa scalar measure of rotation rate, and represents the magnitude of thevector quantity angular velocity. Angular speed or frequency ω, inradians per second, correlates to frequency v in cycles per second, alsocalled Hz, by a factor of 2π. The drive signal includes a drive periodwhere at least one drive pulse is applied to ERM 300, and a monitoringperiod where the back electromagnetic field (“EMF”) of the rotating mass301 is received and used to determine the angular speed of ERM 300. Inanother embodiment, the drive period and the monitoring period areconcurrent and the embodiment of the invention dynamically determinesthe angular speed of ERM 300 during both the drive and monitoringperiods.

FIGS. 4A-4C are views of a piezoelectric implementation of a hapticactuator 18 according to one embodiment of the invention. FIG. 4A showsa disk piezoelectric actuator that includes an electrode 401, a piezoceramics disk 403 and a metal disk 405. As shown in FIG. 4B, when avoltage is applied to electrode 401, the piezoelectric actuator bends inresponse, going from a relaxed state 407 to a transformed state 409.When a voltage is applied, it is that bending of the actuator thatcreates the foundation of vibration. Alternatively, FIG. 4C shows a beampiezoelectric actuator that operates similarly to a disk piezoelectricactuator by going from a relaxed state 411 to a transformed state 413.

FIG. 5 is a view of a haptic device using ESF according to oneembodiment of the invention. The embodiment is based on the discoverythat subcutaneous Pacinian corpuscles can be stimulated by means of acapacitive electrical coupling and an appropriately dimensioned controlvoltage, either without any mechanical stimulation of the Paciniancorpuscles or as an additional stimulation separate from such mechanicalstimulation. An appropriately dimensioned high voltage is used as thecontrol voltage. In the present context, a high voltage means such avoltage that direct galvanic contact must be prevented for reasons ofsafety and/or user comfort. This results in a capacitive couplingbetween the Pacinian corpuscles and the apparatus causing thestimulation, wherein one side of the capacitive coupling is formed by atleast one galvanically isolated electrode connected to the stimulatingapparatus, while the other side, in close proximity to the electrode, isformed by the body member, preferably a finger, of the stimulationtarget, such as the user of the apparatus, and more specifically thesubcutaneous Pacinian corpuscles.

It likely that an embodiment of the invention is based on a controlledformation of an electric field between an active surface of theapparatus and the body member, such as a finger, approaching or touchingit. The electric field tends to give rise to an opposite charge on theproximate finger. A local electric field and a capacitive coupling canbe formed between the charges. The electric field directs a force on thecharge of the finger tissue. By appropriately altering the electricfield a force capable of moving the tissue may arise, whereby thesensory receptors sense such movement as vibration.

As shown in FIG. 5, one or more conducting electrodes 501 are providedwith an insulator. When a body member such as finger 505 is proximate tothe conducting electrode 501, the insulator prevents flow of directcurrent from the conducting electrode to the body member 505. Acapacitive coupling field force 503 over the insulator is formed betweenthe conducting electrode 501 and the body member 505. The apparatus alsocomprises a high-voltage source for applying an electrical input to theone or more conducting electrodes, wherein the electrical inputcomprises a low-frequency component in a frequency range between 10Hz-1000 Hz. The capacitive coupling and electrical input are dimensionedto produce an electrosensory sensation which is produced independentlyof any mechanical vibration of the one or more conducting electrodes orinsulators.

FIG. 6 is a view of a haptic device for inducing acoustic radiationpressure with an ultrasonic haptic transducer. An airborne ultrasoundtransducer array 601 is designed to provide tactile feedback inthree-dimensional (“3D”) free space. The array radiates airborneultrasound, and produces high-fidelity pressure fields onto the user'shands without the use of gloves or mechanical attachments. The method isbased on a nonlinear phenomenon of ultrasound; acoustic radiationpressure. When an object interrupts the propagation of ultrasound, apressure field is exerted on the surface of the object. This pressure iscalled acoustic radiation pressure. The acoustic radiation pressure P[Pa] is simply described as P=αE, where E [J=m³] is the energy densityof the ultrasound and a is a constant ranging from 1 to 2 depending onthe reflection properties of the surface of the object. The equationdescribes how the acoustic radiation pressure is proportional to theenergy density of the ultrasound. The spatial distribution of the energydensity of the ultrasound can be controlled by using the wave fieldsynthesis techniques. With an ultrasound transducer array, variouspatterns of pressure field are produced in 3D free space. Unlikeair-jets, the spatial and temporal resolutions are quite fine. Thespatial resolution is comparable to the wavelength of the ultrasound.The frequency characteristics are sufficiently fine up to 1 KHz.

The airborne ultrasound can be applied directly onto the skin withoutthe risk of the penetration. When the airborne ultrasound is applied onthe surface of the skin, due to the large difference between thecharacteristic acoustic impedance of the air and that of the skin, about99.9% of the incident acoustic energy is reflected on the surface of theskin. Hence, this tactile feedback system does not require the users towear any clumsy gloves or mechanical attachments.

FIG. 7 shows a three-dimensional (“3D”) diagram illustrating a hapticdevice 701 using a haptic substrate and a flexible surface in accordancewith one embodiment of the invention. Device 701 includes a flexiblesurface layer 703, a haptic substrate 705, and a deforming mechanism711. It should be noted that device 701 can be a user interface device,such as an interface for a cellular phone, a personal digital assistant(“PDA”), an automotive data input system, and so forth. It should befurther noted that the underlying concept of the exemplary embodiment ofthe invention would not change if one or more blocks (circuits orlayers) were added to or removed from device 701.

Flexible surface layer 703, in one instance, is made of soft and/orelastic materials such as silicone rubber, which is also known aspolysiloxane. A function of the flexible surface layer 703 is to changeits surface shape or texture upon contact with the physical pattern ofhaptic substrate 705. The physical pattern of haptic substrate 705 isvariable as one or more of the local features of haptic substrate 705can be raised or lowered to present features to affect the surface ofthe flexible surface layer 703 upon contact. Once the physical patternof haptic substrate 705 is determined, the texture of flexible surfacelayer 703 can change to confirm its surface texture to the physicalpattern of haptic substrate 705. It should be note that the deformationof flexible surface layer 703 from one texture to another can becontrolled by deforming mechanism 711. For example, when deformingmechanism 711 is not activated, flexible surface layer 703 maintains itssmooth configuration floating or sitting over haptic substrate 705. Thesurface configuration of flexible surface layer 703, however, deforms orchanges from a smooth configuration to a coarse configuration whendeforming mechanism 711 is activated and the haptic substrate 705 is incontact with the flexible surface layer 703 so as to generate a similarpattern on the top surface of the flexible surface layer 703.

Alternatively, flexible surface layer 703 is a flexible touch sensitivesurface, which is capable of accepting user inputs. The flexible touchsensitive surface can be divided into multiple regions wherein eachregion of the flexible touch sensitive surface can accept an input whenthe region is being touched or depressed by a finger. In one embodiment,the flexible touch sensitive surface includes a sensor, which is capableof detecting a nearby finger and waking up or turning on the device.Flexible surface layer 703 may also include a flexible display, which iscapable of deforming together with flexible surface layer 703. It shouldbe noted that various flexible display technologies can be used tomanufacture flexible displays, such as organic light-emitting diode(“OLED”), organic, or polymer Thin Film Transistor (“TFT”).

Haptic substrate 705 is a surface reconfigurable haptic device capableof changing its surface pattern in response to one or more patternactivating signals. Haptic substrate 705 can also be referred to as ahaptic mechanism, a haptic layer, a tactile element, and the like.Haptic substrate 705, in one embodiment, includes multiple tactile orhaptic regions 707, 709, wherein each region can be independentlycontrolled and activated. Since each tactile region can be independentlyactivated, a unique surface pattern of haptic substrate 705 can becomposed in response to the pattern activating signals. In anotherembodiment, every tactile region is further divided into multiple hapticbits wherein each bit can be independently excited or activated ordeactivated.

Haptic substrate 705, or a haptic mechanism, in one embodiment, isoperable to provide haptic feedback in response to an activating commandor signal. Haptic substrate 705 provides multiple tactile or hapticfeedbacks wherein one tactile feedback is used for surface deformation,while another tactile feedback is used for input confirmation. Inputconfirmation is a haptic feedback to inform a user about a selectedinput. Haptic mechanism 705, for example, can be implemented by varioustechniques including vibration, vertical displacement, lateraldisplacement, push/pull technique, air/fluid pockets, local deformationof materials, resonant mechanical elements, piezoelectric materials,micro-electro-mechanical systems (“MEMS”) elements, thermal fluidpockets, MEMS pumps, variable porosity membranes, laminar flowmodulation, or the like.

Haptic substrate 705, in one embodiment, is constructed by semi-flexibleor semi-rigid materials. In one embodiment, haptic substrate should bemore rigid than flexible surface 703 thereby the surface texture offlexible surface 703 can confirm to the surface pattern of hapticsubstrate 705. Haptic substrate 705, for example, includes one or moreactuators, which can be constructed from fibers (or nanotubes) ofelectroactive polymers (“EAP”), piezoelectric elements, fiber of shapememory alloys (“SMAs”) or the like. EAP, also known as biologicalmuscles or artificial muscles, is capable of changing its shape inresponse to an application of voltage. The physical shape of an EAP maybe deformed when it sustains large force. EAP may be constructed fromElectrostrictive Polymers, Dielectric elastomers, Conducting Polyers,Ionic Polymer Metal Composites, Responsive Gels, Bucky gel actuators, ora combination of the above-mentioned EAP materials.

Shape Memory Alloy (“SMA”), also known as memory metal, is another typeof material which can be used to construct haptic substrate 705. SMA maybe made of copper-zinc-aluminum, copper-aluminum-nickel, nickel-titaniumalloys, or a combination of copper-zinc-aluminum,copper-aluminum-nickel, and/or nickel-titanium alloys. A characteristicof SMA is that when its original shape is deformed, it regains itsoriginal shape in accordance with the ambient temperature and/orsurrounding environment. It should be noted that the present embodimentmay combine the EAP, piezoelectric elements, and/or SMA to achieve aspecific haptic sensation.

Deforming mechanism 711 provides a pulling and/or pushing force totranslate elements in the haptic substrate 705 causing flexible surface703 to deform. For example, when deforming mechanism 711 creates avacuum between flexible surface 703 and haptic substrate 705, flexiblesurface 703 is pushed against haptic substrate 705 causing flexiblesurface 703 to show the texture of flexible surface 703 in accordancewith the surface pattern of haptic substrate 705. In other words, once asurface pattern of haptic substrate 705 is generated, flexible surfaceis pulled or pushed against haptic substrate 705 to reveal the patternof haptic substrate 705 through the deformed surface of flexible surface703. In one embodiment, haptic substrate 705 and deforming mechanism 711are constructed in the same or substantially the same layer.

Upon receipt of a first activating signal, haptic substrate 705generates a first surface pattern. After formation of the surfacepattern of haptic substrate 705, deforming mechanism 711 is subsequentlyactivated to change surface texture of flexible surface 703 in responseto the surface pattern of haptic substrate 705. Alternatively, if hapticsubstrate 705 receives a second activating signal, it generates a secondpattern.

Haptic substrate 705 further includes multiple tactile regions whereineach region can be independently activated to form a surface pattern ofthe substrate. Haptic substrate 705 is also capable of generating aconfirmation feedback to confirm an input selection entered by a user.Deforming mechanism 711 is configured to deform the surface texture offlexible surface 703 from a first surface characteristic to a secondsurface characteristic. It should be noted that haptic device furtherincludes a sensor, which is capable of activating the device when thesensor detects a touch on flexible surface 703. Deforming mechanism 711may be a vacuum generator, which is capable of causing flexible surface703 to collapse against the first surface pattern to transform itssurface configuration in accordance with the configuration of firstpattern of haptic substrate 705.

Haptic substrate 705 illustrates the state when tactile regions 707 and709 are activated. Tactile regions 707 and 709 are raised in a z-axisdirection. Upon receipt of one or more activating signals, hapticsubstrate 705 identifies a surface pattern in accordance with theactivating signals. Haptic substrate 705 provides identified pattern byactivating various tactile regions such as regions 707 and 709 togenerate the pattern. It should be noted that tactile regions 707 and709 imitate two buttons or keys. In another embodiment, tactile region707 or 709 includes multiple haptic bits wherein each bit can becontrolled for activating or deactivating.

FIGS. 8A-8B are views of a haptic device using USF. An ultrasonicvibration display 801 produces ultrasonic vibrations in the order of afew micrometers. The display 801 consists of a touch interface surface803 that vibrates at the ultrasound range. The vibrations 805 travelalong the touch surface 803 at a speed v_(t) when a finger 809 is incontact and applies a force 807 F_(t) to the surface 803. The vibrations805 create an apparent reduction of friction on the surface 803. Oneexplanation is that by moving up and down, the touch surface 803 createsan air gap 813 between the surface 803 and the interacting finger 809,and is the air gap 813 that causes the reduction in friction. This canbe thought as of a Lamb wave 815 along the surface 803 that at someinstants in time is in contact with the finger 809 when the finger 809is in contact with the crest or peak of the wave 805, and sometimes isnot when the finger 809 is above the valley of the wave 805. When finger809 is moved in a lateral direction 811 at a speed v_(f), the apparentfriction of the surface 803 is reduced due to the on and off contact ofthe surface 803 with the finger 809. When the surface 803 is notactivated, the finger 809 is always in contact with the surface 803 andthe static or kinetic coefficients of friction remain constant.

Because the vibrations 805 occur on surface 803 in the ultrasound rangeof typically 20 KHz or greater, the wavelength content is usuallysmaller than the finger size, thus allowing for a consistent experience.It will be noted that the normal displacement of surface 803 is in theorder of less than 5 micrometers, and that a smaller displacementresults in lower friction reduction.

FIG. 9 is an example 5.1 multi-channel audio surround configurationaccording to one embodiment of the invention. The 5.1 multi-channelaudio surround configuration includes a plurality of audio outputdevices, such as speakers. In the illustrated embodiment, the 5.1multi-channel audio surround configuration (i.e., audio output devicesC, Sub, L, R, LS, and RS) that are positioned within a circle thatsurrounds a user U. A source audio signal includes a plurality of sourceaudio channels, where one or more source audio channels are mapped to anaudio output device, where the audio output device output audio effectsbased on the mapped source audio channel(s). According to theembodiment, an LFE channel of the source audio signal can be mapped toaudio output device Sub. As previously described, an LFE channel is anaudio channel that has been encoded with an audio spectrum substantiallyless than the full spectrum of human auditory perception. An LFE trackis typically used to encode low-frequency information with audiblefrequencies in the range of 20 Hz-120 Hz, but may include any otheraudio signal with a limited frequency range. The ‘0.1’ of the 5.1multi-channel audio surround configuration refers to the LFE channel,because the LFE channel typically only requires a fraction of thebandwidth of the other audio channels.

FIG. 10 is human auditory system equal-loudness chart according to oneembodiment of the invention. An equal-loudness chart measures soundpressure (“dB SPL”) over a frequency spectrum, for which a listenerperceives a constant loudness when presented with pure steady tones. Theunit of measurement for loudness levels is the “phon”, and is arrived byreference to equal-loudness contours. A lowest equal-loudness contour1010 represents a quietest audible tone and is also known as theabsolute threshold of hearing. A highest equal-loudness contour 1020 isknown as a threshold of pain.

FIGS. 11A-11B are audio spectrograms showing a shifted, amplified andcompressed audio signal according to one embodiment of the invention.More specifically, FIG. 11A shows an LFE audio signal 1110 that isextracted from a 5.1 audio signal. FIG. 11B shows an LFE audio signal1120 that has been pitch-shifted, compressed, and amplified. Bypitch-shifting an audio signal, one or more original frequencies of theaudio signal are shifted to one or more new frequencies, where the newfrequencies maintain the harmonic relationships of the originalfrequencies, so that a ratio of the new frequencies is the same as aratio of the original frequencies. For example, an audio signal thatincludes a fundamental frequency of 1 KHz and two harmonic frequenciesof 2 KHz and 5 KHz can be pitch-shifted upwards by a factor of 2.5,where the pitch-shifted audio signal includes a fundamental frequency of2.5 KHz and two harmonic frequencies of 5 KHz and 12.5 KHz. In analternate embodiment, LFE audio signal 1120 can be frequency-shifted,rather than pitch-shifted. By frequency-shifting an audio signal, one ormore original frequencies of the audio signal are shifted to one or morenew frequencies, where the new frequencies do not maintain the harmonicrelationships of the original frequencies, so that a ratio of the newfrequencies is not necessarily the same as a ratio of the originalfrequencies. For example, an audio signal that includes a fundamentalfrequency of 1 KHz and two harmonic frequencies of 2 KHz and 5 KHz canbe frequency-shifted upwards by 1.5 KHz, where the frequency-shiftedaudio signal includes a fundamental frequency of 2.5 KHz and twoharmonic frequencies of 3.55 KHz and 6.5 KHz. By either pitch-shifting,or frequency-shifting, LFE audio signal 1110 upwards, one or morefrequencies of LFE audio signal 1110 can be shifted from an originalfrequency range to a target frequency range of a haptic output device,such as an actuator. Because the one or more frequencies of LFE audiosignal 1120 are within a target frequency range of the haptic outputdevice, LFE audio signal 1120 can be a suitable haptic signal for thehaptic output device. In certain embodiments, the original frequencyrange can be a limited frequency range, such as 20 Hz-120 Hz. In theseembodiments, one, some, or all of the one or more frequencies of LFEaudio signal 1110 can be shifted outside of the limited frequency range.Further, in certain embodiments, the target frequency range of thehaptic output device can be determined at run-time based on a type ofthe haptic output device. Examples of a target frequency range caninclude 30 Hz-120 Hz (e.g., for an LRA actuator), and 120 Hz-300 Hz(e.g., for a piezoelectric actuator).

Further, LFE audio signal 1120 can be compressed, amplified, or acombination of the two. Even further, LFE audio signal 1120 can beresampled to a target driving frequency of the haptic output device.More specifically, a rendering frequency of LFE audio signal 1120 can beshifted to a new rendering frequency, where the new rendering frequencycan be equal to a target driving frequency of the haptic output device.This resampling can change how LFE audio signal 1120 is rendered (i.e.,how many samples are played per second) to better fit the capability ofthe haptic output device. The resampling can be performed after LFEaudio signal 1120 is pitch-shifted (or otherwise frequency-shifted), orthe resampling can be performed at run-time, when LFE audio signal 1120is sent to a haptic output device as a haptic signal. Further, in someembodiments, the resampling can be omitted. LFE audio signal 1120 cansubsequently be sent to a haptic output device as a haptic signal, wherethe haptic output device can output one or more haptic effects based onthe haptic signal.

FIGS. 12A-12B are Blackman-Harris windows showing a shifted audio signalaccording to one embodiment of the invention. More specifically, FIG.12A shows an LFE audio signal 1210 that is extracted from a 5.1 audiosignal. FIG. 12B shows an LFE audio signal 1220 that has beenpitch-shifted. By pitch-shifting an audio signal, as previouslydescribed, one or more original frequencies of the audio signal areshifted to one or more new frequencies, where the new frequenciesmaintain the harmonic relationships of the original frequencies, so thata ratio of the new frequencies is the same as a ratio of the originalfrequencies. In an alternate embodiment, LFE audio signal 1220 can befrequency-shifted, rather than pitch-shifted. By frequency-shifting anaudio signal, as previously described, one or more original frequenciesof the audio signal are shifted to one or more new frequencies, wherethe new frequencies do not maintain the harmonic relationships of theoriginal frequencies, so that a ratio of the new frequencies is notnecessarily the same as a ratio of the original frequencies.

FIG. 13 is a quality of experience chart and LFE haptics chart accordingto one embodiment of the invention. According to the embodiment, a studywas conducted with several participants, where test videos were shown tothe participants, where the test videos were displayed on a tabletdevice. Some test videos included haptic effects that were generatedbased on extracting an LFE audio signal from an audio signal containedwithin each video and converting the LFE audio signal into a hapticsignal. Other test videos did not include haptic effects. Theparticipants wore high-quality stereo headphones as they watched thetest videos, and held the tablet device in their hands. The participantswatched the test videos and made real-time ratings with an experiencetime-lining interface. The participants further answered surveyquestions at the end of each test video. An actual presentation order ofthe test videos was varied to prevent repeated showings of the samevideo. The presentation order was counterbalanced between participantsto control for order effects. A total video play time was approximately32-35 minutes, and a total session time was equal to 90 minutes.Further, according to the embodiment, there were a total number of 20participants, where the group of participants was gender-balanced. 50%of the participants indicated that they owned a tablet, and all of theparticipants indicated that they watch media on a smartphone or tabletregularly (i.e., 2-3 times a week).

According to the embodiment, a haptic signal 1310 is a haptic signalthat is generated based on an LFE audio signal that is extracted from anaudio signal included within a first video from the study. An LFEhaptics version of the first video was shown to some of the participantsof the study, where the LFE haptics version of the first video includedhaptic effects generated based on haptic signal 1310. A non-hapticsversion of the first video was shown to other participants of the study,where the non-haptics version of the first video did not include anyhaptic effects. Graph 1320 represents an average quality of experience(“QoE”) rating indicated by the participants over a duration of thenon-haptics version of the first video, where a QoE rating is a ratingfrom 0 to 100 that indicates the quality of the viewing experience,where 0 represents a lowest quality, and where 100 represents a highestquality. Graph 1330 represents an average QoE rating indicated by theparticipants over a duration of the LFE haptics version of the firstvideo. Further, graph 1340 represents a delta of the average QoE ratingof graph 1330 and the average QoE rating of graph 1320 over a durationof the first video. As can be seen from graphs 1320, 1330, and 1340, theparticipants who experienced the LFE haptics version of the first videoindicated a higher rating of quality than the participants whoexperienced the non-haptics version of the first video, especiallyduring portions of the first video where the haptic effects were morepronounced based on the content of haptic signal 1310.

FIGS. 14A-14C illustrate a quality of experience chart and LFE hapticschart according to one embodiment of the invention. A haptic signal 1410is a haptic signal that is generated based on an LFE audio signal thatis extracted from an audio signal included within a second video fromthe study. An LFE haptics version of the second video was shown to someof the participants of the study, where the LFE haptics version of thesecond video included haptic effects generated based on haptic signal1410. A non-haptics version of the second video was shown to otherparticipants of the study, where the non-haptics version of the secondvideo did not include any haptic effects. Graph 1420 represents anaverage QoE rating indicated by the participants over a duration of thenon-haptics version of the second video. Graph 1430 represents anaverage QoE rating indicated by the participants over a duration of theLFE haptics version of the second video. Further, graph 1440 representsa delta of the average QoE rating of graph 1430 and the average QoErating of graph 1420 over a duration of the second video. As can be seenfrom graphs 1420, 1430, and 1440, the participants who experienced theLFE haptics version of the second video indicated a higher rating ofquality than the participants who experienced the non-haptics version ofthe second video, especially during portions of the second video wherethe haptic effects were more pronounced based on the content of hapticsignal 1410.

FIGS. 15A and 15B illustrate a quality of experience chart andcorresponding LFE haptics chart according to one embodiment of theinvention. A haptic signal 1510 is a haptic signal that is generatedbased on an LFE audio signal that is extracted from an audio signalincluded within a third video from the study. An LFE haptics version ofthe third video was shown to some of the participants of the study,where the LFE haptics version of the third video included haptic effectsgenerated based on haptic signal 1510. A non-haptics version of thethird video was shown to other participants of the study, where thenon-haptics version of the third video did not include any hapticeffects. Graph 1520 represents an average QoE rating indicated by theparticipants over a duration of the non-haptics version of the thirdvideo. Graph 1530 represents an average QoE rating indicated by theparticipants over a duration of the LFE haptics version of the thirdvideo. Further, graph 1540 represents a delta of the average QoE ratingof graph 1530 and the average QoE rating of graph 1520 over a durationof the third video. As can be seen from graphs 1520, 1530, and 1540, theparticipants who experienced the LFE haptics version of the third videoindicated a higher rating of quality than the participants whoexperienced the non-haptics version of the third video, especiallyduring portions of the third video where the haptic effects were morepronounced based on the content of haptic signal 1510.

FIGS. 16A and 16B illustrate a quality of experience chart andcorresponding LFE haptics chart according to one embodiment of theinvention. A haptic signal 1610 is a haptic signal that is generatedbased on an LFE audio signal that is extracted from an audio signalincluded within a fourth video from the study. An LFE haptics version ofthe fourth video was shown to some of the participants of the study,where the LFE haptics version of the fourth video included hapticeffects generated based on haptic signal 1610. A non-haptics version ofthe fourth video was shown to other participants of the study, where thenon-haptics version of the fourth video did not include any hapticeffects. Graph 1620 represents an average QoE rating indicated by theparticipants over a duration of the non-haptics version of the fourthvideo. Graph 1630 represents an average QoE rating indicated by theparticipants over a duration of the LFE haptics version of the fourthvideo. Further, graph 1640 represents a delta of the average QoE ratingof graph 1630 and the average QoE rating of graph 1620 over a durationof the fourth video. As can be seen from graphs 1620, 1630, and 1640,the participants who experienced the LFE haptics version of the fourthvideo indicated a higher rating of quality than the participants whoexperienced the non-haptics version of the fourth video, especiallyduring portions of the fourth video where the haptic effects were morepronounced based on the content of haptic signal 1610.

FIGS. 17A-17C illustrate a quality of experience chart and correspondingLFE haptics chart according to one embodiment of the invention. A hapticsignal 1710 is a haptic signal that is generated based on an LFE audiosignal that is extracted from an audio signal included within a fifthvideo from the study. An LFE haptics version of the fifth video wasshown to some of the participants of the study, where the LFE hapticsversion of the fifth video included haptic effects generated based onhaptic signal 1710. A non-haptics version of the fifth video was shownto other participants of the study, where the non-haptics version of thefifth video did not include any haptic effects. Graph 1720 represents anaverage QoE rating indicated by the participants over a duration of thenon-haptics version of the fifth video. Graph 1730 represents an averageQoE rating indicated by the participants over a duration of the LFEhaptics version of the fifth video. Further, graph 1740 represents adelta of the average QoE rating of graph 1730 and the average QoE ratingof graph 1720 over a duration of the fifth video. As can be seen fromgraphs 1720, 1730, and 1740, the participants who experienced the LFEhaptics version of the fifth video indicated a higher rating of qualitythan the participants who experienced the non-haptics version of thefifth video, especially during portions of the fifth video where thehaptic effects were more pronounced based on the content of hapticsignal 1710.

FIG. 18 is an immersiveness summary chart according to one embodiment ofthe invention. The immersiveness summary chart compares immersiveratings indicated by participants of the study for the LFE hapticsversions of the videos with immersive ratings indicated by participantsof the study for the non-haptics versions of the videos, where animmersive rating is a rating from 0 to 9 that indicates how immersedinto the viewing experience the participant was, where 0 represents alowest amount of immersiveness, and where 9 represents a highest amountof immersiveness.

The immersiveness summary chart of FIG. 18 includes immersiveness ratingsets 1810, 1815, 1820, 1825, 1830, 1835, 1840, 1845, 1850, and 1855.Immersiveness rating set 1810 represents immersiveness ratings for anon-haptics version of a first video of the study. Immersiveness ratingset 1815 represents immersiveness ratings for an LFE haptics version ofthe first video. An average rating of immersiveness rating set 1815(i.e., 6.15) is higher than an average rating of immersiveness ratingset 1810 (i.e., 4.9). Further, immersiveness rating set 1820 representsimmersiveness ratings for a non-haptics version of a second video of thestudy. Immersiveness rating set 1825 represents immersiveness ratingsfor an LFE haptics version of the second video. An average rating ofimmersiveness rating set 1825 (i.e., 6.7) is higher than an averagerating of immersiveness rating set 1820 (i.e., 5.2). Immersivenessrating set 1830 represents immersiveness ratings for a non-hapticsversion of a third video of the study. Immersiveness rating set 1835represents immersiveness ratings for an LFE haptics version of the thirdvideo. An average rating of immersiveness rating set 1835 (i.e., 6.7) ishigher than an average rating of immersiveness rating set 1830 (i.e.,5.4).

Further, immersiveness rating set 1840 represents immersiveness ratingsfor a non-haptics version of a fourth video of the study. Immersivenessrating set 1845 represents immersiveness ratings for an LFE hapticsversion of the fourth video. An average rating of immersiveness ratingset 1845 (i.e., 6.55) is higher than an average rating of immersivenessrating set 1840 (i.e., 4.65). Immersiveness rating set 1850 representsimmersiveness ratings for a non-haptics version of a fifth video of thestudy. Immersiveness rating set 1855 represents immersiveness ratingsfor an LFE haptics version of the fifth video. An average rating ofimmersiveness rating set 1855 (i.e., 6.9) is higher than an averagerating of immersiveness rating set 1850 (i.e., 5.15). Thus, alldifferences between the immersiveness ratings for the LFE hapticsversions of the videos and the immersive ratings for the non-hapticsversions of the videos are statistically significant. Further, onaverage, the LFE haptics versions of the videos were rated 30% moreimmersive by the participants of the study, as compared with thenon-haptics versions of the videos.

FIG. 19 is a quality of experience summary chart according to oneembodiment of the invention. The quality of experience summary chartcompares QoE ratings indicated by participants of the study for the LFEhaptics versions of the videos with QoE ratings indicated byparticipants of the study for the non-haptics versions of the videos,where a QoE rating is a rating from 0 to 100 that indicates the qualityof the viewing experience, where 0 represents a lowest quality, andwhere 100 represents a highest quality.

The quality of experience summary chart of FIG. 19 includes QoE ratingsets 1910, 1915, 1920, 1925, 1930, 1935, 1940, 1945, 1950, and 1955. QoErating set 1910 represents QoE ratings for a non-haptics version of afirst video of the study. QoE rating set 1915 represents QoE ratings foran LFE haptics version of the first video. An average rating of QoErating set 1915 (i.e., 70.4) is higher than an average rating of QoErating set 1910 (i.e., 58.6). Further, QoE rating set 1920 representsQoE ratings for a non-haptics version of a second video of the study.QoE rating set 1925 represents QoE ratings for an LFE haptics version ofthe second video. An average rating of QoE rating set 1925 (i.e., 74.1)is higher than an average rating of QoE rating set 1920 (i.e., 62.5).QoE rating set 1930 represents QoE ratings for a non-haptics version ofa third video of the study. QoE rating set 1935 represents QoE ratingsfor an LFE haptics version of the third video. An average rating of QoErating set 1935 (i.e., 74.5) is higher than an average rating of QoErating set 1930 (i.e., 63.9).

Further, QoE rating set 1940 represents QoE ratings for a non-hapticsversion of a fourth video of the study. QoE rating set 1945 representsQoE ratings for an LFE haptics version of the fourth video. An averagerating of QoE rating set 1945 (i.e., 75.1) is higher than an averagerating of QoE rating set 1940 (i.e., 60). QoE rating set 1950 representsQoE ratings for a non-haptics version of a fifth video of the study. QoErating set 1955 represents QoE ratings for an LFE haptics version of thefifth video. An average rating of QoE rating set 1955 (i.e., 75) ishigher than an average rating of QoE rating set 1950 (i.e., 58.9). Thus,all differences between the QoE ratings for the LFE haptics versions ofthe videos and the QoE ratings for the non-haptics versions of thevideos are statistically significant. Further, on average, the LFEhaptics versions of the videos were rated 21% higher by the participantsof the study, as compared with the non-haptics versions of the videos.

FIG. 20 is a flow diagram for converting an LFE audio signal into ahaptic signal according to an embodiment of the invention. In oneembodiment, the functionality of the flow diagram of FIG. 20, as well asthe functionality of the flow diagrams of FIGS. 21, 22, 23, 24, 25, 28,29, and 30 are each implemented by software stored in memory or othercomputer readable or tangible medium, and executed by a processor. Inother embodiments, each functionality may be performed by hardware(e.g., through the use of an application specific integrated circuit(“ASIC”), a programmable gate array (“PGA”), a field programmable gatearray (“FPGA”), etc.), or any combination of hardware and software. Incertain embodiments, some functionality may be omitted.

The flow begins and proceeds to 2001. At 2001, a source audio signal isreceived. The source audio signal includes multiple audio signals, wherethe audio signals can be contained within multiple tracks or channels.At least one of the multiple audio signals can be an LFE audio signal,where the LFE audio signal is contained within an LFE track or LFEchannel. The flow then proceeds to 2003.

At 2003, the source audio signal is down-mixed to a surround encodingwhich includes the LFE audio signal. In certain embodiments, thesurround encoding can be a 5.1 surround encoding. The flow then proceedsto 2005.

At 2005, the LFE audio signal is extracted from the source audio signal.In certain embodiments where the LFE audio signal is encoded within thesource audio signal, the extracted LFE audio signal is also decoded. Theflow then proceeds to 2007.

At 2007, the LFE audio signal is converted into a haptic signal. Incertain embodiments, the LFE audio signal can be converted into thehaptic signal by pitch-shifting the LFE audio signal. By pitch-shiftingthe LFE audio signal, an original pitch of the LFE audio signal can beshifted to a target pitch within a target pitch range of a haptic outputdevice, such as an actuator. Further, by shifting the original pitch ofthe LFE audio signal to the target pitch, one or more originalfrequencies of the LFE audio signal can be shifted to one or more targetfrequencies, where a ratio of the one or more target frequencies is thesame as a ratio of the one or more original frequencies. In certainembodiments, the original pitch of the LFE audio signal can be within alimited pitch range, such as 20 Hz-100 Hz. In some embodiments, theshift of the original pitch of the LFE audio signal is a shift of theoriginal pitch outside of the limited pitch range. In some of theseembodiments, the shift of the original pitch of the LFE audio signal isa shift of the original pitch completely outside of the limited pitchrange. In other embodiments, the LFE audio signal can be converted intothe haptic signal by frequency-shifting the LFE audio signal. Byfrequency-shifting the LFE audio signal, one or more originalfrequencies of the LFE audio signal can be shifted to one or more targetfrequencies within a target frequency range, where a ratio of the one ormore target frequencies is not the same as a ratio of the one or moreoriginal frequencies. In certain embodiments, the one or more originalfrequencies of the LFE audio signal can be within a limited frequencyrange, such as 20 Hz-100 Hz. In some embodiments, the shift of the oneor more original frequencies of the LFE audio signal is a shift of theone or more original frequencies outside of the limited frequency range.In some of these embodiments, the shift of the one or more originalfrequencies of the LFE audio signal is a shift of the one or moreoriginal frequencies completely outside of the limited frequency range.The flow then proceeds to 2009.

At 2009, the haptic signal is resampled to a target driving frequency ofthe haptic output device. An example target driving frequency is 8 KHz.The flow then proceeds to 2011.

At 2011, the haptic signal is encoded in a container, or streaming dataformat, of a file which supports haptic data encoding. In certainembodiments, the haptic signal is encoded within the LFE audio signal.The haptic signal can subsequently be extracted from the container,decoded, and sent to the haptic output device, where the haptic signalcauses the haptic output device to output one or more haptic effects.The flow then ends.

In certain embodiments, one or more parameters may be provided (eitherautomatically or by a user) to adjust the pitch-shift, orfrequency-shift, to control an amount or locality of the pitch-shift, orfrequency-shift. Further, in some embodiments, additional processing ofthe LFE audio signal can be performed before or after the pitch-shift,or frequency-shift. The additional processing can include filtering, orother “smoothing” operations, to remove noise introduced by thepitch-shift, of frequency-shift. Further, in some embodiments, thehaptic signal can be sent to the haptic output device in real-time ornear real-time. Even further, in some embodiments, the one or moreparameters may be provided by the user using an authoring tool, such asa digital audio authoring software application.

FIG. 21 is a flow diagram for converting an LFE audio signal into ahaptic signal according to another embodiment of the invention. The flowbegins and proceeds to 2101. At 2101, a source audio signal is received.The source audio signal includes multiple audio signals, where the audiosignals can be contained within multiple tracks or channels. At leastone of the multiple audio signals can be an LFE audio signal, where theLFE audio signal is contained within an LFE track or LFE channel. Theflow then proceeds to 2103.

At 2103, the source audio signal is spatialized in real-time to asurround encoding which includes the LFE audio signal. In certainembodiments, the surround encoding can be a 5.1 surround encoding. Theflow then proceeds to 2105.

At 2105, the LFE audio signal is extracted from the source audio signal.In certain embodiments where the LFE audio signal is encoded within thesource audio signal, the extracted LFE audio signal is also decoded. Theflow then proceeds to 2107.

At 2107, the LFE audio signal is converted into a haptic signal. Incertain embodiments, the LFE audio signal can be converted into thehaptic signal by pitch-shifting the LFE audio signal, as previouslydescribed in conjunction with FIG. 20. In other embodiments, the LFEaudio signal can be converted into the haptic signal byfrequency-shifting the LFE audio signal, as previously described inconjunction with FIG. 20. The flow then proceeds to 2109.

At 2109, the haptic signal is resampled to a target driving frequency ofa haptic output device, such as an actuator. An example target drivingfrequency is 8 KHz. The flow then proceeds to 2111.

At 2111, the haptic signal is sent to the haptic output device inreal-time, where the haptic signal causes the haptic output device tooutput one or more haptic effects. The flow then ends.

FIG. 22 is a flow diagram for converting an LFE audio signal into ahaptic signal according to another embodiment of the invention. The flowbegins and proceeds to 2201. At 2201, a source audio signal is received.The source audio signal includes multiple audio signals, where the audiosignals can be contained within multiple tracks or channels. At leastone of the multiple audio signals can be an LFE audio signal, where theLFE audio signal is contained within an LFE track or LFE channel of thesource audio signal. The flow then proceeds to 2203.

At 2203, the source audio signal is decoded into multiple audio signals,where the multiple audio signals includes the LFE audio signal. The flowthen proceeds to 2205.

At 2205, the LFE audio signal is extracted from the source audio signal.In certain embodiments where the LFE audio signal is encoded within thesource audio signal, the extracted LFE audio signal is also decoded. Theflow then proceeds to 2207.

At 2207, the LFE audio signal is converted into a haptic signal. Incertain embodiments, the LFE audio signal can be converted into thehaptic signal by pitch-shifting the LFE audio signal, as previouslydescribed in conjunction with FIG. 20. In other embodiments, the LFEaudio signal can be converted into the haptic signal byfrequency-shifting the LFE audio signal, as previously described inconjunction with FIG. 20. The flow then proceeds to 2209.

At 2209, the haptic signal is resampled to a target driving frequency ofa haptic output device, such as an actuator. An example target drivingfrequency is 8 KHz. The flow then proceeds to 2211.

At 2211, the haptic signal is sent to the haptic output device inreal-time, where the haptic signal causes the haptic output device tooutput one or more haptic effects. The flow then ends.

FIG. 23 is a flow diagram for encoding a haptic signal within alow-frequency effect signal according to an embodiment of the invention.Encoding haptic signals in audio signals can be convenient becausecontent for two different outputs can be recorded, stored, andtransmitted in a single waveform, or other type of signal. In the caseof an LFE audio signal, an audio signal in a limited frequency range(e.g., 20 Hz-120 Hz) is stored and played back at an audio outputdevice, such as a speaker, with the same frequency range. Thus,according to an embodiment, a haptic signal can be encoded within anyband-limited frequency range of an LFE audio signal, such as a highfrequency range (e.g., greater than 200 Hz), without interfering withaudio data contained within the LFE audio signal. A design of hapticeffects that are generated based on the haptic signal can be done in anormal frequency range, such as 1 Hz-200 Hz, but, at a time of encodingthe haptic signal in the LFE audio signal, the haptic signal can beshifted to any band-limited frequency range, so it can be stored withoutinterfering with the audio data of the LFE audio signal. One advantageof this encoding is that the haptic signal can be derived directly fromthe LFE audio signal, or authored by a haptic effect developer, and thehaptic signal can be encoded in the same LFE audio signal.

The flow begins and proceeds to 2301. At 2301, an audio signal iscreated that includes audio data. The audio signal also includes an LFEaudio signal. The LFE audio signal can include audio data with one ormore frequencies within a limited frequency range, such as 20 Hz-120 Hz.The flow then proceeds to 2303.

At 2303, a haptic signal is created that includes haptic data. Thehaptic data can include one or more frequencies within a normalfrequency range, such as 1 Hz-200 Hz. The flow then proceeds to 2305.

At 2305, the haptic signal is encoded within the LFE audio signal of theaudio signal. In certain embodiments, the haptic signal can be encodedwithin a target frequency range of the LFE audio signal. The targetfrequency range of the LFE audio signal can be any band-limitedfrequency range, such as a high frequency range (e.g., greater than 200Hz). In these embodiments, one or more frequencies of the haptic signalcan be shifted from one or more frequencies within the normal frequencyrange to one or more frequencies within the band-limited frequencyrange, before the haptic signal is encoded within the band-limitedfrequency range of the LFE audio signal. In one embodiment, the audiosignal can be a Digital Dolby audio signal, and thus, the haptic signalcan be encoded with the LFE audio signal of the Digital Dolby audiosignal. The flow then proceeds to 2307.

At 2307, the audio signal is either stored within a container orstreaming data format of a file, or is transmitted. The flow then ends.

FIG. 24 is a flow diagram for decoding a haptic signal from alow-frequency effect signal according to an embodiment of the invention.As previously described, a design of haptic effects that are generatedbased on the haptic signal can be done in a normal frequency range, but,at a time of encoding the haptic signal in the LFE audio signal, thehaptic signal can be shifted to any band-limited frequency range, so itcan be stored without interfering with the audio data of the LFE audiosignal. At a time of decoding, a filtering technique can be used toextract the LFE audio signal from an audio signal, and to furtherextract the haptic signal from the LFE audio signal. The filteringtechnique can further be used to frequency-shift the haptic signal fromthe band-limited frequency range to the normal frequency range.

The flow begins and proceeds to 2401. At 2401, it is determined whethera device that performs decoding has haptic support (i.e., whether thedevice can output haptic effects). If the device does not have hapticsupport, the flow proceeds to 2403. If the device has haptic support,the flow proceeds to 2405.

At 2403, the audio signal that includes the LFE audio signal is playedat an audio output device. In certain embodiments, the audio signal issent to the audio output device, where the audio signal causes the audiooutput device to output one or more audio effects. The flow then ends.

At 2405, the haptic signal is extracted from the LFE audio signal of theaudio signal and decoded. In certain embodiments, the haptic signal isextracted from a target frequency range of the LFE audio signal. Thetarget frequency range of the LFE audio signal can be any band-limitedfrequency range, such as a high-frequency range (e.g., greater than 200Hz). In these embodiments, one or more frequencies of the haptic signalcan be shifted from one or more frequencies within the band-limitedfrequency range to one or more frequencies within a normal frequencyrange, such as 1 Hz-200 Hz, after the haptic signal is decoded. The flowthen proceeds to 2407.

At 2407, the haptic signal is played at a haptic output device. Incertain embodiments, the haptic signal is sent to the haptic outputdevice, where the haptic signal causes the haptic output device tooutput one or more haptic effects. The flow then proceeds to 2409.

At 2409, the audio signal that includes the LFE audio signal is playedat an audio output device. In certain embodiments, the audio signal issent to the audio output device, where the audio signal causes the audiooutput device to output one or more audio effects. The flow then ends.

FIG. 25 is a flow diagram for converting an LFE audio signal into aplurality of haptic signals according to an embodiment of the invention.The flow begins and proceeds to 2501. At 2501, a source audio signal isreceived. The source audio signal includes multiple audio signals, wherethe audio signals can be contained within multiple tracks or channels.At least one of the multiple audio signals can be an LFE audio signal,where the LFE audio signal is contained within an LFE track or LFEchannel. The flow then proceeds to 2503.

At 2503, the LFE audio signal is extracted from the source audio signal.In certain embodiments where the LFE audio signal is encoded within thesource audio signal, the extracted LFE audio signal is also decoded. Theflow then proceeds to 2505.

At 2505, the LFE audio signal is converted into a plurality of hapticsignals. In certain embodiments, the conversion of the LFE audio signalinto the haptic signals can be sequential. In other embodiments, theconversion of the LFE audio signal into the haptic signals can besimultaneous. Further, in certain embodiments, the LFE audio signal canbe converted into each haptic signal by pitch-shifting the LFE audiosignal. By pitch-shifting the LFE audio signal, an original pitch of theLFE audio signal can be shifted to a target pitch within a target pitchrange of a haptic output device, such as an actuator. There can be aplurality of haptic output devices, and each haptic output device canhave a distinct target pitch range. Thus, each pitch-shift of the LFEaudio signal can shift the original pitch of the LFE audio signal to atarget pitch within each distinct target pitch range of each hapticoutput device. In other embodiments, the LFE audio signal can beconverted into each haptic signal by frequency-shifting the LFE audiosignal. By frequency-shifting the LFE audio signal, one or more originalfrequencies of the LFE audio signal can be shifted to one or more targetfrequencies within a target frequency range of a haptic output device,such as an actuator. There can be a plurality of haptic output devices,and each haptic output device can have a distinct target frequencyrange. Thus, each frequency-shift of the LFE audio signal can shift theone or more original frequencies of the LFE audio signal to one or moretarget frequencies within each distinct target frequency range of eachhaptic output device. The flow then proceeds to 2507.

At 2507, the haptic signals are sent to the haptic output devices inreal-time, where each haptic signal causes the corresponding hapticoutput device to output one or more haptic effects. In an alternateembodiment, each haptic signal is encoded in a container, or streamingdata format, of a file which supports haptic data encoding. In certainembodiments, at least one haptic signal is encoded within the LFE audiosignal. Each haptic signal can subsequently be extracted from thecontainer, decoded, and sent to the corresponding haptic output device,where each haptic signal causes the corresponding haptic output deviceto output one or more haptic effects. The flow then ends.

FIGS. 26A-26D are screen views of example foreground and backgroundhaptic applications according to one embodiment of the invention. Itwill be recognized that more than one haptic enabled softwareapplication may be running simultaneously on a device having a hapticactuator, and that a window on the top of a virtual windows environmentmay overlap or obscure portions of any windows that are on the bottom.FIG. 26A shows a screen view of an example application window having avirtual download application button located in the center of the screen.In FIG. 26B the user selects the download application button, whereuponFIG. 26C shows a new screen view having a status bar in the center ofthe screen which indicates the percentage completion of the download.The status bar changes color proportionally from left to rightcorresponding to the percentage completion text shown directly below thestatus bar. Because the status bar is haptified, a haptic effect signalis generated and output to the haptic actuator concurrently with thevisual display of the status bar. In one embodiment, the haptic effectsignal changes over time corresponding to the percentage completion ofthe download.

FIG. 26D shows a screen view of a text input window. The text inputwindow, selected by the user as the active window, is running in theforeground and completely obscures the download application status barwhich is running simultaneously in the background. Although the downloadapplication window is no longer the active window and the status bar iscompletely obscured on the visual display, the status bar haptic effectsignal continues to be generated and output to the haptic actuator as abackground haptic effect. Because the text input window is alsohaptified, a foreground haptic effect signal is generated and output tothe haptic actuator for each typed character concurrently with thevisual display of the typed character in the text input window. In oneembodiment, the foreground and background haptic effect signals arecombined, modified or synthesized in such a way that the user perceivesthe foreground and background haptic effects as being distinct hapticeffects even though they are both being output concurrently via a singlehaptic actuator.

The perception of a haptic effect has three different levels. The firstlevel is the threshold of perception, which is the minimum appliedhaptic effect signal component or components required for a user todetect the haptic effect. Such haptic components include, but are notlimited to, strength, frequency, duration, rhythm and dynamics of thehaptic effect signal. It will be recognized that the threshold of hapticperception may be highly non-linear and may vary greatly between users,and may even vary for a single user depending on many factors such asthe user's sensitivity to touch, how tightly the user may be holding ahandheld device, the ambient temperature, the user's age, or the user'sphysical activity or environment such as walking or riding in a vehicle,and so on.

The second level of haptic effect perception is the threshold ofattention break-in, which is the minimum change in the applied hapticeffect signal that results in drawing the user's attention away from theprimary focus to the attention break-in haptic effect itself. It will berecognized that the threshold of attention break-in may vary betweenusers or for a single user depending on many factors as described above,and may also vary depending on whether the attention break-in is relatedto various types of haptic effects including a positive additive effect,or a negative subtractive effect, or a change in the haptic effect. Thethird level of haptic effect perception is the threshold of pain, whichalso varies between users or for a single user depending on many factorsas described above. It will be recognized that under some circumstances,the threshold of perception may be the same as the threshold ofattention break-in, which may also be the same as the threshold of pain.

Embodiments of the invention are compatible with a wide variety ofhaptic actuators, and can present multiple channels of haptic effectdata with different intensity levels. In one embodiment, the multiplechannels are represented by a foreground channel and one or morebackground channels. A background haptic effect is any haptic effect orhaptic effect component which meets or exceeds the threshold ofperception. A foreground haptic effect is any haptic effect or hapticeffect component which meets or exceeds the threshold of attentionbreak-in. In one embodiment, a foreground or background haptic effectmay be a defined set of static or dynamic haptic effects or effectcomponents. In another embodiment, a foreground or background hapticeffect may be an adaptive set of static or dynamic haptic effects orhaptic effect components in response to user input, system input, devicesensor input or ambient input.

Using multiple haptic channels, such as foreground and backgroundchannels, enables subtle haptic effects to be provided concurrently withmore obvious haptic effects, allowing a user to distinguish between thedifferent effects and identifying them as originating from differentsources. In one embodiment, low-importance or high-density informationis perceivable, but not overwhelming or distracting from a primary task,and multiple channels further enable haptic ambient awareness. Forexample, a haptic enabled handheld or mobile device which is monitoringthe local weather during a rainstorm activates a background hapticchannel to provide a sensation of raindrops that increases or decreasesas it rains harder or softer.

In one embodiment, foreground and background channels are used todistinguish the feedback originating from a local device and thefeedback originating from another user. For example, a messagenotification arriving from another user activates a foreground hapticeffect, while the status of a ticking clock on the local deviceactivates a background haptic effect.

In one embodiment, foreground and background channels are used todistinguish the feedback originating from a local device and thefeedback originating from a primary user. For example, the feedbackoriginated by a primary user typing on a haptic enabled keyboardactivates a foreground haptic effect, while the status of a progress baron the local device activates a background haptic effect.

In one embodiment, foreground and background channels are used todistinguish the feedback within or between virtual simulations oranimations. For example, the motion of a virtual rolling ball activatesa foreground haptic effect, while the virtual texture the ball isrolling on activates a background haptic effect.

In one embodiment, background haptic effects are additive such that whenmultiple background effects are received concurrently or in quicksuccession, the overall result is a natural or gradual foregrounding ofthe haptic effects. For example, a single background text message“tweet” notification received from a non-primary user may be easilymissed or ignored by the primary user, but when hundreds or thousands ofmessage notifications constituting a “tweet storm” are received in ashort amount of time, the multiple haptic effects add up and the overallresult is a haptic experience in the foreground which draws the primaryuser's attention to the event.

In one embodiment, background haptic effects are used to providenon-distracting or “polite” augmentation of a commercial advertisementor any other type of haptic encoded content. For example, anadvertisement for a carbonated soft drink provides a background haptic“fizz” effect that can be felt if the user is paying attention butotherwise can be easily ignored.

It will be recognized that any type of input such as user, device,system, application or network input may be represented by any number ofhaptic events on one or more foreground or background haptic channels.Examples include, but are not limited to, multi-tasking applications,incoming email, “tweet” message notifications, passive notifications,outgoing messages, progress bars, Bluetooth or local device pairings,network add or drop connection, continuous antenna signal level, and soon.

FIGS. 27A-27B are display graphs of example multiple data channels ofhaptic feedback according to one embodiment of the invention. FIG. 27Ashows a graph of the perceptual magnitude of a haptic signal over timefor priority based haptic events, along with a corresponding graph ofnotification activity. At time T1, the perceptual magnitude of a hapticsignal 2701 corresponding to the medium priority notifications N1 and N2starts in the background channel 2703. Upon receipt of a high prioritynotification N3, at time T2 the haptic signal 2701 begins to rise untilat time T3 the haptic signal 2701 crosses the threshold from thebackground channel 2703 into the foreground channel 2705. The hapticsignal 2701 continues to increase up to a peak level 2707, where in theabsence of any further notifications the haptic signal 2701 decreasesand crosses the threshold from the foreground channel 2705 to thebackground channel 2703 at time T4.

At time T5, receipt of a high priority notification once again causesthe haptic signal 2701 to rise until at time T6 the haptic signal 2701crosses the threshold from the background channel 2703 into theforeground channel 2705. The haptic signal 2701 continues to increase upto a peak level 2709, where in the absence of any further notificationsthe haptic signal 2701 decreases and crosses the threshold from theforeground channel 2705 to the background channel 2703 at time T7. Itwill be recognized that a stream of low-priority or medium-prioritynotifications punctuated with high-priority notifications results in ahaptic signal 2701 that shifts between the background channel 2703 andforeground channel 2705 without limitation.

FIG. 27B shows a graph of the perceptual magnitude of a haptic signalover time for frequency based haptic events, along with a correspondinggraph of notification activity. At time T8, the perceptual magnitude ofa haptic signal 2711 corresponding to the relatively infrequentnotifications N1 through N3 starts in the background channel 2713. Uponreceipt of higher frequency notifications, at time T9 the haptic signal2711 begins to rise until at time T10 the haptic signal 2711 crosses thethreshold from the background channel 2713 into the foreground channel2715. With continuing receipt of higher frequency notifications, thehaptic signal 2711 continues to increase up to a peak level 2717, wherein the absence of any further notifications the haptic signal 2711decreases and crosses the threshold from the foreground channel 2715 tothe background channel 2713 at time T11. It will be recognized that astream of low-frequency notifications punctuated with high-frequencynotifications results in a haptic signal 2711 that shifts between thebackground channel 2713 and foreground channel 2715 without limitation.In one embodiment, priority based haptic events and frequency basedhaptic events may be interspersed with each other or received at anytime or in any order, and may be used in any manner to generate anoverall combined haptic signal.

FIG. 28 is a flow diagram for displaying multiple data channels ofhaptic feedback for priority based haptic events according to oneembodiment of the present invention. At 2801, the system receives inputof first and second haptic effect signals having first and secondpriority levels. It will be recognized that any type or number ofpriority levels may be used, such as foreground and background prioritylevels, or any number of alpha-numeric or any other sequential ornon-sequential priority levels, without limitation. The first and secondhaptic effect signals may be received in any order or time sequence,either sequentially with non-overlapping time periods or in parallelwith overlapping or concurrent time periods. At 2803, the systemcompares the first priority level to the second priority level. If at2805 the first priority level is less than the second priority level, at2807 an interaction parameter is generated using the second hapticsignal. It will be recognized that any type of input synthesis methodmay be used to generate the interaction parameter from one or morehaptic effect signals including, but not limited to, the method ofsynthesis examples listed in TABLE 1 below. If at 2809 the firstpriority level is equal to the second priority level, at 2811 aninteraction parameter is generated using the second haptic signal. If at2813 the first priority level is greater than the second priority level,at 2815 an interaction parameter is generated using the second hapticsignal. At 2817, a drive signal is applied to a haptic actuatoraccording to the interaction parameter.

TABLE 1 METHODS OF SYNTHESIS Additive synthesis - combining inputs,typically of varying amplitudes Subtractive synthesis - filtering ofcomplex signals or multiple signal inputs Frequency modulationsynthesis - modulating a carrier wave signal with one or more operatorsSampling - using recorded inputs as input sources subject tomodification Composite synthesis - using artificial and sampled inputsto establish a resultant “new” input Phase distortion - altering thespeed of waveforms stored in wavetables during playback Waveshaping -intentional distortion of a signal to produce a modified resultResynthesis - modification of digitally sampled inputs before playbackGranular synthesis - combining of several small input segments into anew input Linear predictive coding - similar technique as used forspeech synthesis Direct digital synthesis - computer modification ofgenerated waveforms Wave sequencing - linear combinations of severalsmall segments to create a new input Vector synthesis - technique forfading between any number of different input sources Physical modeling -mathematical equations of the physical characteristics of virtual motion

FIG. 29 is a flow diagram for displaying multiple data channels ofhaptic feedback for frequency based haptic events according to oneembodiment of the present invention. At 2901, the system receives one ormore haptic effect notifications N over a non-zero time period T. At2903, the system generates a notification frequency ratio R, calculatedby using at least the number of haptic effect notifications N and thenon-zero time period T. In one embodiment, the notification frequencyratio R is calculated as N divided by T. At 2905, the system comparesthe notification frequency ratio R to a foreground haptic threshold F.Haptic threshold F may be static or dynamic and may vary over timedepending on many factors such as the user's sensitivity to touch, howtightly the user may be holding a handheld device, the ambienttemperature, the user's age, or the user's physical activity orenvironment such as walking or riding in a vehicle, and so on. It willbe recognized that the notification frequency ratio R may be directlycalculated or may be normalized corresponding to a wide range ofvariation for the haptic threshold F, and that the haptic threshold Fmay be directly calculated or may be normalized corresponding to a widerange of variation for the notification frequency ratio R.

If at 2907 the notification frequency ratio R is less than theforeground haptic threshold F, at 2909 an interaction parameter isgenerated using a background haptic signal. If at 2911 the notificationfrequency ratio R is greater than or equal to the foreground hapticthreshold F, at 2913 an interaction parameter is generated using aforeground haptic signal. At 2915, a drive signal is applied to a hapticactuator according to the interaction parameter.

FIG. 30 is a flow diagram for converting an LFE audio signal into ahaptic signal according to another embodiment of the invention. The flowbegins and proceeds to 3001. At 2501, an LFE audio signal is received.The flow then proceeds to 3003.

At 3003, the LFE audio signal is converted into a haptic signal. Incertain embodiments, the LFE audio signal is transformed, and thetransformed LFE audio signal can be used as the haptic signal. The flowthen proceeds to 2507.

At 3005, either: (a) the haptic signal is sent to a haptic outputdevice, where the haptic signal causes the haptic output device tooutput one or more haptic effects; or (b) the haptic signal is encodedin a container, or streaming data format, of a file which supportshaptic data encoding. In certain embodiments where the haptic signal isencoded, the haptic signal is encoded within the LFE audio signal.Further, in embodiments where the haptic signal is encoded, the hapticsignal can subsequently be extracted from the container, decoded, andsent to the haptic output device, where the haptic signal causes thehaptic output device to output one or more haptic effects. The flow thenends.

Thus, in one embodiment, a system can extract an LFE audio signal from asource audio signal, and convert the LFE audio signal into a hapticsignal. The haptic signal can then be sent to a haptic output device,such as an actuator, where the haptic signal can cause the haptic outputdevice to output one or more haptic effects. The system can coordinatethe output of the one or more haptic effects with the output of thesource audio signal, which can result is an enhanced experience from aperspective of a user, who experiences both the audio content and thehaptic content. Further, by only converting a component of the sourceaudio signal (i.e., the LFE audio signal), rather than the entire sourceaudio signal, the system is less computationally intensive than systemsthat convert the entire source audio signal. Further, in anotherembodiment, the system can encode the haptic signal within the LFE audiosignal of the audio signal. Using the LFE audio signal can reduce thecomplexity of an overall architecture to encode, store, transmit, anddecode haptic signals. The only addition to the audio signal is thehaptic data added to a band-limited frequency range of the LFE audiosignal of the audio signal, such as a high frequency range. However,this additional data does not affect the LFE audio data, since the audiooutput devices typically do not have the capability of outputting theband-limited frequency data, such as high-frequency data. Therefore,this encoding can be backward-compatible with non-haptic playbackdevices. Another advantage of the encoding is that the same audioeffects designer can design the LFE audio effects as well as the hapticeffects in the same signal. The signal can subsequently be distributedand played back accordingly.

The features, structures, or characteristics of the invention describedthroughout this specification may be combined in any suitable manner inone or more embodiments. For example, the usage of “one embodiment,”“some embodiments,” “certain embodiment,” “certain embodiments,” orother similar language, throughout this specification refers to the factthat a particular feature, structure, or characteristic described inconnection with the embodiment may be included in at least oneembodiment of the present invention. Thus, appearances of the phrases“one embodiment,” “some embodiments,” “a certain embodiment,” “certainembodiments,” or other similar language, throughout this specificationdo not necessarily all refer to the same group of embodiments, and thedescribed features, structures, or characteristics may be combined inany suitable manner in one or more embodiments.

One having ordinary skill in the art will readily understand that theinvention as discussed above may be practiced with steps in a differentorder, and/or with elements in configurations which are different thanthose which are disclosed. Therefore, although the invention has beendescribed based upon these preferred embodiments, it would be apparentto those of skill in the art that certain modifications, variations, andalternative constructions would be apparent, while remaining within thespirit and scope of the invention. In order to determine the metes andbounds of the invention, therefore, reference should be made to theappended claims.

We claim:
 1. A method comprising: generating a low frequency informationaudio signal based on a multichannel audio signal; synthesizing a hapticsignal from the low frequency information audio signal; whereinsynthesizing the haptic signal comprises synthesizing, from the lowfrequency information audio signal, one or more haptic effectparameters; providing the one or more haptic effect parameters to a userfor adjusting the one or more haptic effect parameters; receiving hapticeffect metadata reflecting one or more user adjustments to at least oneof the one or more haptic effect parameters; embedding the haptic signaland the haptic effect metadata in a multichannel audio codec bit streamthat encodes at least the multichannel audio signal; wherein the methodis performed by a computing device.
 2. The method of claim 1, whereingenerating the low frequency information audio signal comprisesextracting a low frequency effects audio channel signal from themultichannel audio signal.
 3. The method of claim 1, wherein generatingthe low frequency information audio signal comprises generating acomposite audio signal based on a right channel of the multichannelaudio signal, a left channel of the multichannel audio signal, a leftsurround channel of the multichannel audio signal, a right surroundchannel of the multichannel audio signal, and a low frequency effectschannel of the multichannel audio signal.
 4. The method of claim 1,wherein one of the one or more haptic effect parameters is frequency. 5.The method of claim 1, wherein one of the one or more haptic effectparameters is magnitude.
 6. The method of claim 1, wherein one of theone or more haptic effect parameters is a frequency modulation synthesishaptic effect parameter that modifies the haptic signal based on one ormore operators.
 7. The method of claim 1, wherein embedding the hapticsignal comprises compressing the haptic signal to produce a compressedhaptic signal and embedding the compressed haptic signal in themultichannel audio codec bit stream.
 8. The method of claim 1, whereinthe haptic signal is embedded in the multichannel audio codec bit streamas a target channel.
 9. The method of claim 1, wherein providing the oneor more haptic effect parameters to the user further comprises providingthe one or more haptic effect parameters to a tool; and whereinreceiving the haptic effect metadata reflecting the one or more useradjustments further comprises receiving the haptic effect metadatareflecting the one or more user adjustments made using the tool.
 10. Themethod of claim 1, wherein the tool is a digital audio authoringsoftware application.
 11. The method of claim 1, wherein embedding thehaptic signal in a multichannel audio codec bit stream further comprisesencoding the haptic signal in a low-frequency effects track of a DolbyDigital audio signal.
 12. The method of claim 1, further comprising:encoding the haptic signal in a container; and embedding the containerincluding the haptic signal in the multichannel audio code bit stream.13. The method of claim 1, further comprising: embedding the hapticsignal in a streaming data format of the multichannel audio codec bitstream.
 14. A non-transitory computer-readable medium storinginstructions which, when executed by a computing device, causeperformance of a method as recited in claim
 1. 15. A method comprising:receiving a multichannel audio codec bitstream comprising a haptic trackand associated haptic effect metadata; decoding the multichannel audiocodec bitstream to produce a multichannel audio signal; sending themultichannel audio signal to one or more audio output devices; decodingthe haptic track to produce a first haptic signal; generating a secondhaptic signal based on the associated haptic effect metadata; modulatingthe first haptic signal with the second haptic signal to produce a thirdhaptic signal; sending the third haptic signal to a haptic actuator;wherein the method is performed by one or more computing devices.
 16. Anon-transitory computer-readable medium storing instructions which, whenexecuted by a computing device, cause performance of a method as recitedin claim
 15. 17. A computing device comprising: a memory configured tostore a low-frequency effects conversion module; a processor configuredto execute the low-frequency effects conversion module stored on thememory; wherein the low-frequency effects conversion module isconfigured, when executed by the processor, to generate a low frequencyinformation audio signal based on a multichannel audio signal; whereinthe low-frequency effects conversion module is further configured, whenexecuted by the processor, to synthesize a haptic signal from the lowfrequency information audio signal; wherein the low-frequency effectsconversion module is further configured, when executed by the processor,to synthesize, from the low frequency information audio signal, one ormore haptic effect parameters; wherein the low-frequency effectsconversion module is further configured, when executed by the processor,to provide the one or more haptic effect parameters to a user foradjusting the one or more haptic effect parameters; wherein thelow-frequency effects conversion module is further configured, whenexecuted by the processor, to receive haptic effect metadata reflectingone or more user adjustments to at least one of the one or more hapticeffect parameters; and wherein the low-frequency effects conversionmodule is further configured, when executed by the processor, to embedthe haptic signal and the haptic effect metadata in a multichannel audiocodec bit stream that encodes at least the multichannel audio signal.18. The computing device of claim 17, wherein the low-frequency effectsconversion module is further configured, when executed by the processor,to extract a low frequency effects audio channel signal from themultichannel audio signal.
 19. The computing device of claim 17, whereinthe low-frequency effects conversion module is further configured, whenexecuted by the processor, to generate a composite audio signal based ona right channel of the multichannel audio signal, a left channel of themultichannel audio signal, a left surround channel of the multichannelaudio signal, a right surround channel of the multichannel audio signal,and a low frequency effects channel of the multichannel audio signal.20. The computing device of claim 17, wherein one of the one or morehaptic effect parameters is frequency.